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o ' 

' Abstract. The purpose of the present mostly expository paper (based mainly on [16, 

18, 39, 17, 12]) is to present the current state of the following conjecture of A. Fel'shtyn 
' and R. Hill [13], which is a generalization of the classical Burnside theorem. 

Let G be a countable discrete group, (p one of its automorphisms, the number 

of (/)-conjugacy (or twisted conjugacy) classes, and S'(0) = # Fix((/)) the number of 0- 
invariant equivalence classes of irreducible unitary representations. If one of R{4>) and 
' S{4i) is finite, then it is equal to the other. 

This conjecture plays a important role in the theory of twisted conjugacy classes (see 
[25], [10]) and has very important consequences in Dynamics, while its proof needs rather 
sophisticated results from Functional and Non-commutative Harmonic Analysis. 

First we prove this conjecture for finitely generated groups of type I and discuss its 
applications. 

After that we discuss an important example of an automorphism of a type Hi group 
which disproves the original formulation of the conjecture. 
^ ' Then we prove a version of the conjecture for a wide class of groups, including almost 

, polycyclic groups (in particular, finitely generated groups of polynomial growth). In this 

formulation the role of an appropriate dual object plays the finite-dimensional part of 
the unitary dual. Some counter-examples are discussed. 

Then we begin a discussion of the general case (which also needs new definition of the 
dual object) and prove the weak twisted Burnside theorem for general countable discrete 
groups. For this purpose we prove a non-commutative version of Riesz-Markov-Kakutani 
\ representation theorem. 

Finally we explain why the Reidemeister numbers are always infinite for Baumslag- 
Solitar groups. 
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1. Introduction and formulation of results 

Definition 1.1. Let G* be a countable discrete group and : G ^ G an endomorphism. 
Two elements x,x' E G are said to be (p- conjugate or twisted conjugate, iff there exists 
g E G with 

x' = gx(f){g~^). 

We shall write {x}^ for the (p-conjugacy or twisted conjugacy class of the element x E G. 
The number of 0-conjugacy classes is called the Reidemeister number of an endomorphism 
(j) and is denoted by R{4>)- If (f) is the identity map then the ^-conjugacy classes are the 
usual conjugacy classes in the group G. 

If G is a finite group, then the classical Burnside theorem (see e.g. [26, p. 140]) says that 
the number of classes of irreducible representations is equal to the number of conjugacy 
classes of elements of G. Let G be the unitary dual of G, i.e. the set of equivalence classes 
of unitary irreducible representations of G. 

Remark 1.2. If : G — G is an epimorphism, it induces a map (j) : G ^ G, (f){p) = pocp 
(because a representation is irreducible if and only if the scalar operators in the space of 
representation are the only ones which commute with all operators of the representation). 
This is not the case for a general endomorphism 0, because pcj) can be reducible for 
an irreducible representation p, and can be defined only as a multi-valued map. But 
nevertheless we can define the set of fixed points Fix0 of on G. 

Therefore, by the Burnside's theorem, if is the identity automorphism of any finite 
group G, then we have -R(0) = #Fix(0). 

To formulate the main theorem of the first part of the paper for the case of a general 
endomorphism we first need an appropriate definition of the Fix(0). 

Definition 1.3. Let Rep(G) be the space of equivalence classes of finite dimensional 
unitary representations of G. Then the corresponding map 0/j : Rep(G) Rep(G) is 
defined in the same way as above: 0r(p) = p o 0. 

Let us denote by Fix(0) the set of points p G G C Rep(G) such that 0r(p) = p. 

Theorem 1.4. Let G he a finitely generated discrete group of type \, one of its endo- 
morphism, R{(p) the number of (p-conjugacy classes, and 5'(0) = # Fix(0) the number of 
(p-invariant equivalence classes of irreducible unitary representations. If one of R{(p) and 
S{(p) is finite, then it is equal to the other. 
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Let fi{d), d EN,he the Mohius function, i.e. 

( I if = 1, 
= < (—1)'^ if d is a product of k distinct primes, 
[^0 if (i is not square — free. 

Theorem 1.5 (Congruences for the Reidemeister numbers). Let (p : G ^ G be an endo- 
morphism of a countable discrete group G such that all numbers R{(j)"') are finite and let 
H be a subgroup of G with the properties 

4>{H) C H, 

yx e G 3n e N such that </)"(x) G H. 

If the pair {H, 0") satisfies the conditions of Theorem 1.4 for any n eN, then one has for 
all n, 

= modn. 

d\n 

These theorems were proved previously in a special case of Abelian finitely generated 
plus finite group [13, 14]. 

For groups of type IIi the situation is much more complicated. We discuss in detail the 
case of a semi-direct product of the action of Z on Z © Z by a hyperbolic automorphism 
with finite Reidemeister number (four to be precise) and the number of fixed points of 
on G equal or greater than five [18]. The origin of this phenomenon lies in bad separation 
properties of G for general discrete groups. A more deep study leads to the following 
general theorem. 

Theorem 1.6 (weak twisted Burnside theorem, [39]). The number R^,{(j)) of Reidemeister 
classes related to twisted invariant functions on G from the Fourier- Stieltjes algebra B{G) 
is equal to the number Stf{(j)) of generalized fixed points of (p on the Glimm spectrum of G, 
i. e. on the complete regularization ofG, if one of R^{(j)) and S^{(j)) is finite. 

The argument goes along the following line. The well-known Riesz(-Markov-Kakutani) 
theorem identifies the space of linear functionals on algebra A = G{X) and the space 
of regular measures on X. To prove the weak twisted Burnside theorem we first ob- 
tain a generalization of this theorem to the case of non-commutative C*-algebra A via 
Dauns-Hofmann sectional representation theorem. The corresponding measures on Glimm 
spectrum are functional- valued. In extreme situation this theorem is tautological, but for 
group C*-algebras of discrete groups in many cases one obtains some new tool for counting 
twisted conjugacy classes. 

Keeping in mind that for hyperbolic groups -R(0) is always infinite while in the "op- 
posite" case the twisted Burnside theorem is proved, we can formulate the following 
conjecture, which is in fact a program of further actions. 

Conjecture 1.7. There exists a class of groups Q such that 

• for any group G ^ Q and any automorphism cj) : G ^ G the Reidemeister number 
R{(p) is always infinite, 
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• for any group G E Q there exists a subset of ideals Ai C Prim C*{G) such that its 
points are separated and R{(f)) coincides with the number of fixed points of on 
A4 supposing one of these numbers to be finite. 

One of candidates for Ai is the set of maximal ideals (let us remind that the Glimm 
spectrum is in fact the space of maximal ideals of the center of C*{G), cf. [4, 39]). The 
strategy of proof will be based on the weak theorem 1.6. 

We also consider a formulation of the main conjecture with counting only finite-dimen- 
sional fixed points on the unitary dual. In [17] we prove this version for a very large class 
of groups. 

Theorem 1.8. For a wide class of groups, which includes almost polycyclic groups {in 
particular, finitely generated groups of polynomial growth) the Reidemeister number R{(j)) 
coincides with the number 5/(0) of finite- dimensional (p- fixed points on the unitary dual, 
if R{4>) is finite. ^ 

This version of the main conjecture (with the consideration of only finite-dimensional 
representations) has some counter-examples. One of them, coming from D. Osin's group 
[32] we discuss after the mentioned theorem in Section 11. Also we propose there some 
possible formulation of the conjecture appropriate for residually finite groups. 

Let us remark, that in some cases the weak twisted Burnside theorem easily implies the 
twisted Burnside theorem (in particular, in the form with finite-dimensional representa- 
tions). For example, we will show directly that R{4>) = R*{4>) in Abelian case. On the 
other hand, the unitary dual coincides with the Glimm spectrum. Slightly more compli- 
cated argument is valid for some more general groups covered by Theorem 1.8, in particu- 
lar, the Heisenberg group. More precisely, it is possible to extract from the first part of the 
proof of Theorem 1.8 (see below Section 11) that -R(0) = R*{(f))- Moreover, characteristic 
functions of Reidemeister classes are related to finite dimensional representations, which 
are maximal and Hausdorff separated. Hence, the finite-dimensional fixed points form a 
part of generalized fixed points on Glimm spectrum and S^:{(f)) > Sf{(j)). In fact, we have 
an equality here, because if there exists a functional coming from some other generalized 
fixed point on Glimm spectrum, it cannot be a coefficient of a finite-dimensional rep- 
resentation because Glimm spectrum is Hausdorff separated while all finite-dimensional 
(generalized) fixed points are already counted in 5/(0). The details concerning the rela- 
tion between separateness and linear independence are contained in Section 11. 

The interest in twisted conjugacy relations has its origins, in particular, in the Nielsen- 
Reidemeister fixed point theory (see, e.g. [25, 10]), in Selberg theory (see, eg. [37, 1]), 
and Algebraic Geometry (see, e.g. [22]). 



After the acceptance of the present paper for pubhcation, we obtained a more strong version of this 
theorem using another method. Namely, the end of the theorem sounds as "if one of these numbers is 
finite" . For this purpose we introduce the notion of (^-conjugacy separable group (for any two (/)-conjugacy 
classes there exists a 0-respecting homomorphism of G onto a finite group such that the images of these 
classes do not intersect). A sufficient condition for this property is conjugacy separability of the semidirect 
product of G and Z by (j). By the famous result of Rcmeslcnikov and Formanek this is the case for any 
almost polycyclic group. 



REIDEMEISTER CLASSES AND TWISTED BURNSIDE THEOREM 



5 



The congruences give some necessary conditions for the reahzation problem for Rei- 
demeister numbers in topological dynamics. The relations with Selberg theory will be 
presented in a forthcoming paper. 

Let us remark that it is known that the Reidemeister number of an endomorphism of 
a finitely generated Abelian group is finite iff 1 is not in the spectrum of the restriction 
of this endomorphism to the free part of the group (see, e.g. [25]). The Reidemeister 
number is infinite for any automorphism of a non-elementary Gromov hyperbolic group 



To make the presentation more detailed and transparent we start from a new approach 
(E.T.) for Abelian (Section 2) and compact (Section 3) groups. Only after that we develop 
this approach and prove the main theorem for finitely generated groups of type I [16] in 
Section 5. A discussion of some examples leading to conjectures is the subject of Section 6. 
In Section 7 we present the mentioned important example for type IIi groups [18]. After 
that we pass to the demonstration of the weak twisted Burnside theorem in Section 10. For 
this purpose we present in Section 9 a non-commutative version of Riesz-Markov-Kakutani 
theorem [39] (some necessary information on operator fields is collected in Section 8). In 
Section 11 we discuss almost polycyclic groups and related matter. Section 12 is devoted 
to the discussion of Baumslag-Solitar groups [12]. 
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Let be an automorphism of an Abelian group G. 
Lemma 2.1. The twisted conjugacy class H of e is a subgroup. The other ones are cosets 



Lemma 2.2. Suppose, ui,U2 G G, xh is the characteristic function of H as a set. Then 



[11]. 



2. Abelian case 




h-^(f){h). 



□ 
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Proof. Suppose, ui G giH, U2 G g2H, hence, ui = gihi, U2 = (72^2- Then 

uiU2^ = gihih^^g:^^ G gig2^H. 

Thus, xh{uiU2^) = 1 if and only if gig2^ G H and ui and U2 are in the same class. 
Otherwise it is 0. □ 

The following Lemma is well known. 

Lemma 2.3. For any subgroup H the function xh is of positive type. 

Proof. Let us take arbitrary elements mi,M2, ■ ■ ■ ,Un of G. Let us reenumerate them in 
such a way that some first are in giH, the next ones are in g2H, and so on, till gmH, 
where gjH are different cosets. By the previous Lemma the matrix \\pit\\ '■= \\XH{'>J'iU'j~^)\\ 
is block-diagonal with square blocks formed by units. These blocks, and consequently the 
whole matrix are positively semi-defined. □ 

Lemma 2.4. In the Abelian case characteristic functions of twisted conjugacy classes 
belong to the Fourier- Stieltjes algebra B{G) = {C*{G))*. 

Proof. By Lemma 2.1 in this case the characteristic functions of twisted conjugacy classes 
are the shifts of the characteristic function of the class H of e. Hence, by Corollary (2.19) 
of [7], these characteristic functions are in B{G). □ 

Let us remark that there exists a natural isomorphism (Fourier transform) 

u^u^ C*(G)=C;(G)-C(G), g{p):=p{g), 

(this is a number because irreducible representations of an Abelian group are 1-dimen- 
sional). In fact, it is better to look (for what follows) at an algebra G{G) as an algebra 
of continuous sections of a bundle of 1-dimensional matrix algebras, over G. 

Our characteristic functions, being in B{G) = {G*{G))* in this case, are mapped to the 
functionals on G{G) which, by the Riesz-Markov-Kakutani theorem, are measures on G. 
Which of these measures are invariant under the induced (twisted) action of G ? Let us 
remark, that an invariant non-trivial functional gives rise to at least one invariant space 
-its kernel. 

Let us remark, that convolution under the Fourier transform becomes point-wise mul- 
tiplication. More precisely, the twisted action, for example, is defined as 

g[f]{p) = p{g)f{p)p{<l>{g-')), peG, geG, feG{G). 

There are 2 possibilities for the twisted action of G on the representation algebra 
Ap = C: 1) the linear span of the orbit of 1 G is equal to all Ap, 2) and the opposite 
case (the action is trivial). 

The second case means that the space of intertwining operators between Ap and A^^ 

equals C, and p is a fixed point of the action (f) : G G. In the first case this is the 
opposite situation. 

If we have a finite number of such fixed points, then the space of twisted invariant 
measures is just the space of measures concentrated in these points. Indeed, let us describe 
the action of G on measures in more detail. 

Lemma 2.5. For any Borel set E one has g[fj]{E) = J^g[l] dfi. 
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Proof. The restriction of measure to any Borel set commutes with the action of G, since 
the last is point wise on C{G). For any Borel set E one has 



g[fi]{E) = / ldg[fi]= / g[l]dfx. 

J E J E 



□ 

Hence, if /i is twisted invariant, then for any Borel set E and any g & G one has 



/ {l-g[l])dfi = 0. 

J E 



Lemma 2.6. Suppose, f G C{X), where X is a compact Hausdorff space, and fi is a 
regular Borel measure on X, i.e. a functional on C{X). Suppose, for any Borel set 
E <Z X one has j^f djji = 0. Then fi{h) = for any h G C{X) such that f{x) = 
implies h{x) = 0. I.e. fi is concentrated off the interior o/suppf. 

Proof. Since the functions of the form fh are dense in the space of the referred to above 
/i's, it is sufficient to verify the statement for fh. Let us choose an arbitrary e > and a 

n 

simple function h' = o-iXE^ such that \^{fh') — fi{fh) \ < e. Then 

i=l 

n „ n „ 

f^ifh') = ^ aifdfi = '^ai f d/i = 0. 

i=i i=i 

Since e is an arbitrary one, we are done. □ 

Applying this lemma to a twisted invariant measure fi and f = 1 — g[l] we obtain that 
H is concentrated at our finite number of fixed points of 0, because outside of them / 7^ 0. 

If we have an infinite number of fixed points, then the space is infinite-dimensional (we 
have an infinite number of measures concentrated in finite number of points, each time 
different) and Reidemeister number is infinite as well. So, we are done. 

3. Compact case 

Let G be a compact Hausdorff group, hence G is a discrete space. Then C*{G) = ©Mj, 
where Mj are the matrix algebras of irreducible representations. The infinite sum is in 
the following sense: 

C*{G) = {/,}, z G {1, 2, 3, ...} = G, /. G M„ ||/,|| ^ 0(z ^ 00). 

When G is finite and G is finite this is exactly Peter- Weyl theorem. 

A characteristic function of a twisted class is a functional on C*{G). For a finite group 
it is evident, for a general compact group it is necessary to verify only the measurability 
of the twisted class with the respect to Haar measure, i.e. that twisted class is Borel. For 
a compact G, the twisted conjugacy classes being orbits of twisted action are compact 
and hence closed. 

Under the identification it passes to a sequence {ipi}, where ipi is a functional on Mi 
(the properties of convergence can be formulated, but they play no role at the moment). 
The conditions of invariance are the following: for each pj G G one has g[(pi] = ipi, i.e. for 
any a E Mi and any g E G one has (pi{pi{g)api{(f){g~^))) = ipi{a). 

Let us recall the following well-known fact. 
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Lemma 3.1. Each functional on matrix algebra has form a t— > Tr(a6) for a fixed matrix 
b. 

Proof. One has dim(M(?7,, C))' = dim(M(n, C)) = n x n and looking at matrices as at 
operators in V, dim^ = n, with base e^, one can remark that functionals a {aei,ej), 
i, j = 1, . . . ,n, are hnearly independent. Hence, any functional takes form 

a t-^ ^b'j{aei,ej) = ^6}a^ = Tr(6a), b := 

□ 

Now we can study invariant functionals: 

Tr{bpi{g)api{(f){g~^))) = Tr(6a), Va,5(, 
TT{{b - pi{(j){g~^))bpi{g))a) = 0, Va,^, 

hence, 

b-pM9-'))bp:{g) = 0, yg. 

Since pi is irreducible, the dimension of the space of such 6 is 1 if pi is a fixed point of (p 
and in the opposite case by the Schur lemma. So, we are done. 

Remark 3.2. In fact we are only interested in finite discrete case. Indeed, for a compact 
G, the twisted conjugacy classes being orbits of twisted action are compact and hence 
closed. If there is a finite number of them, then they are open as well. Hence, the situation 
is more or less reduced to a discrete group: quotient by the component of unity. 

4. Extensions and Reidemeister classes 

Let us denote hj Tg : G ^ G the automorphism Tg(g) = gg g~^ for g ^ G. Its restriction 
on a normal subgroup we will denote by well. 

Lemma 4.1. {g}<j,k = {fi- A;}t-^_,o0- 

Proof. Let g' = f g (pif"^) be 0-conjugate to g. Then 

g'k = fg k = fgkk-' <p{f~') k = f{gk) (r,-i o <p) (/" i). 

Conversely, if g' is r^-i o 0-conjugate to g, then 

g'k-' = fg{T,-. o <j>){f~')k-' = fgk-' <j>{f-'). 

Hence a shift maps </)-conjugacy classes onto classes related to another automorphism. □ 
Corollary 4.2. R{4>) = R{Tg o 0). 

Consider a group extension respecting homomorphism 0: 

(1) 




where if is a normal subgroup of G. The argument below, especially related the role of 
fixed points, has a partial intersection with [20, 19]. 
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First of all let us notice that the Reidemeister classes of in G are mapped epimorphi- 
cally on classes of (f) in G/H. Indeed, 

(2) P{9)P{9)4>{P{9-^)) = P(99<P{r')- 

Suppose, R{4>) < oo. Then the previous remark implies R{(f>) < oo. Consider a class 
D = {h}rg^i, where Tg{h) := ghg^^, g E G, h E H. The corresponding equivalence 
relation is 

(3) h ^hhg(f)'(h-^)g-\ 

Since H is normal, the automorphism Tg : H ^ H is well defined. We will denote by D 
the image iD as well. By (3) the shift Dg is a subset of Hg is characterized by 

(4) hgr^hihg)(l)'(h~'). 

Hence it is a subset of {hg}^!, fl Hg and the partition Hg = ^{{h}rg(f,')g is a subpartition 
of Hg = U{Hgn{hg}^). 

We need the following statements. 

Lemma 4.3. Suppose, the extension (1) satisfies the following conditions: 

(1) #Fix0 = k < oo, 

(2) R{(f)) < oo. 

Then 

(5) R{^') <k-{R{^)-R($) + l). 

If G/H is ahelian, let gi he some elements withp{gi) being representatives of all different 
(f)-conjugacy classes, i = 1, . . . , R{(f)). Then 

R&) 

(6) ^i?(r,^0')<A;-W)- 

i=l 

Proof. Consider classes {z}tf)>, z E G, i.e. the classes of relation z ~ hz(j)'{h~^), h E H. 
The group G acts on them hj z ^ gz(f){g~^). Indeed, 

g[hhct^Ch~'mg~') = {ghg~'){ghct^{g~')mg)^Ch~')ct^{9^')) 

= {ghg-'){gh<P{g-'))<P{ghg~^) E {gh<P{g-')}^,, 

because H is normal and ghg~^ G H . Due to invertibility, this action of G transposes 
classes {2;}</,' inside one class {g}^. Hence, the number d of classes {/i},/,' inside {h}^ fl H 
does not exceed the number oi g E G such that p{g)(p{p{g~^)) = e. Since two elements 
g and gh in one i7-coset induce the same permutation of classes {h}^', the mentioned 
number d does not exceed the number of 2; E G/H such that z(f){z~^) = e, i.e. d < k. 
This implies (5). 

Now we discuss 0-classes over 0-classes other than {e}^ for an abelian G/H. An estima- 
tion analogous to the above one leads to the number of 2; e G/H such that zzq(I){z~^) = zq 
for some fixed Zq. But for an Abelian G/H they form the same group Fix(0). This to- 
gether with the description (4) of shifts of D at the beginning of the present Section 
implies (6). □ 
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Lemma 4.4. Suppose, in the extension (1) the group H is abelian. Then # Fix(0) < 
#Fix(</>')-#Fix(0). 

Proof. Let s : G/H — > G be a section of p. If s{z)h is a fixed point of tlien 

(7) {siz)r'<P{s{z)) = h<P'ih~'). 

Hence, z G Fix(0) and left hand side takes k := #Fix(0) values hi,...,hk. Let us 
estimate the number of s{z)h for a fixed z such that {s{z))~^(j){s{z)) = hi. These h have 
to satisfy (7). Since H is abelian, if one has 

hi = h(p\h~^) =h(f)'(h-^), 

then h~^h G Fix(0') and we are done. □ 

Theorem 4.5. Let A be a finitely generated Abelian group, i/) : A ^ A its automorphism 
with R{ip) < oo. Then j^Fix{ip) < oo. 
Moreover, R{ip) > ^Fix{ijj). 

Proof. Let T be the torsion subgroup. It is finite and characteristic. We obtain the 
extension T ^ A ^ A/T respecting cj). Since A/T ^Z'', Fix(V^ : A/T ^ A/T) = e, by 
[25], [10, Sect. 2.3.1]. Hence, by Lemma 4.4, #Fix(V') < #Fix(?^')> ^' :T ^T. For any 
finite abelian group T one clearly has i^Fix{ilj') = R{ip') by Theorem 11.3 (cf. [10, p. 7]). 
Finally, < R{ij) by (6). □ 

Lemma 4.6. Suppose, \G/H\ = N < oo. Then R{Tg(f)') < NR{(j)). More precisely, the 
mentioned subpartition is not more than in N parts. 

Proof. Consider the following action of G on itself: x i— *• gx(f){g~^). Then its orbits are 
exactly classes {x}^. Moreover it maps classes (4) onto each other. Indeed, 

gh{hg)<f>'(h-')<p{g-') = hg{hg)<p{g-')<P'(h-') 

using normality of the H. This map is invertible (g ^ g^^), hence bijection. Moreover, g 
and gh, for any h & H, act in the same way. Or in the other words, H is in the stabilizer 
of this permutation of classes (4). Hence, the cardinality of any orbit < N. □ 

Hence, for any finite G/H the number of classes of the form (4) is finite: it is < NR{(j)). 

Lemma 4.7. Suppose, H satisfies the following property: for any automorphism of H 
with finite Reidemeister number the characteristic functions of Reidemeister classes of 
are linear combinations of matrix elements of some finite number of irreducible finite di- 
mensional representations of H. Then the characteristic functions of classes (4) are linear 
combinations of matrix elements of some finite number of irreducible finite dimensional 
representations of G. 

Proof. Let pi, p2, • • • , Pfc be the above irreducible representations of H, p its direct sum act- 
ing on V, and vr the regular (finite dimensional) representation of G/H. Let p{, . . . , pj., p^ 
be the corresponding induced representations of G. Let us remind that in this sim- 
ple situation the representation p^ is defined as a representation of G in the space 
kiG/H, V) ^ ©f /^'y defined by the formula 

Wi9)f]ix) = p{s{x)g{s{xg)r^)f{xg), f G kiG/H, V), x G G/H, 
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for some fixed section s : G/H ^ G of the canonical projection G G/H. 

Let the characteristic function of D be represented under the form Xoih) = {p{h)^,ri). 
Let e L'^{G/H, V) be defined by the formulas ^^(e) =^eV, ^\g) = Oifg=^e. Define 
similarly rj^ . Then for h G i{H) we have 

p'{h)e{9) = p{s{g)hs{9hr'm9h) = p{hs{g)s{g)-')m = 

Hence, {p\h)^^ ,r]^)\i(^H) is the characteristic function of i{D). Let u,v G L'^{G/H) be 
such vectors that {n{'g)u, v) is the characteristic function of e. Then 

is the characteristic function of i{D). Other characteristic functions of classes (4) are 
shifts of this one. Hence, they are matrix elements of the representation p^ (8> vr. It 
is finite dimensional. Hence it can be decomposed in a finite direct sum of irreducible 
representations. □ 

Corollary 4.8 (of previous two lemmata). Under the assumptions of the previous lemma, 
the characteristic functions of Reidemeister classes of (p are linear combinations of matrix 
elements of some finite number of irreducible finite dimensional representations of G. 

5. The case of groups of type I 

Theorem 5.1. Let G be a discrete group of type I. Then 

• [6, 3.L4, 4.L11] The dual space G is a Ti -topological space. 

• [38] Any irreducible representation of G is finite- dimensional. 

Remark 5.2. In fact a discrete group G is of type I if and only if it has a normal, Abelian 
subgroup M of finite index. The dimension of any irreducible representation of G is at 
most [G : M] [38]. 

Suppose R = R{4>) < oo, and let F C L°°{G) be the i?-dimensional space of all 
twisted-invariant functionals on L^{G). Let K C L}{G) be the intersection of kernels of 
functionals from F. Then is a linear subspace of L^{G) of codimension R. For each 
p G (j let us denote by Kp the image p{K). This is a subspace of a (finite-dimensional) 
full matrix algebra. Let cdp be its codimension. 

Let us introduce the following set 

Gf = {peG \ cdp^ 0}. 
Lemma 5.3. One has cdp if and only if p is a fixed point of (p. In this case cdp = 1. 

Proof. Suppose, cdp ^ and let us choose a functional (fp on the (finite-dimensional full 
matrix) algebra p{L^{G)) such that Kp C Kei (pp. Then for the corresponding functional 
(p*p = ipp o p on L^{G) one has K C Kercyj*. Hence, ip*p E F and is twisted-invariant, as 
well as if p. Then we argue as in the case of compact group (text after Lemma 3.1). 

Conversely, if p is a fixed point of 0, it gives rise to a (unique up to scaling) non-trivial 
twisted-invariant functional (pp. Let x = p{a) be any element in p{L}{G)) such that 
^Pp{x) 7^ 0. Then x ^ Kp, because p*p{a) = 'Pp{x) ^ 0, while is a twisted-invariant 
functional on L^((T). So, cdp ^ 0. 



1 0, otherwise. 
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The uniqueness (up to scaling) of the intertwining operator imphes the uniqueness of 
the corresponding twisted- invariant functional Hence, cdp = 1. □ 

Hence, 

(8) Gf = Fix(0). 

From the property cdp = 1 one obtains for this (unique up to scahng) functional ipp-. 

(9) KeT^p = Kp. 
Lemma 5.4. R = H^Gp, in particular, the set Gp is finite. 

Proof. First of all we remark that since G is finitely generated almost Abelian (cf. Remark 
5.2) there is a normal Abelian subgroup H of finite index invariant under (p. Hence we can 
apply Lemma 4.8 to G, H, (p. So there is a finite collection of irreducible representations 
of G such that any twisted-invariant functional is a linear combination of matrix elements 
of them, i.e. linear combination of functionals on them. If one of them gives a non-trivial 
contribution, it has to be a twisted-invariant functional on the corresponding matrix 
algebra. Hence, by the argument above, these representations belong to Gp, and the 
appropriate functional is unique up to scaling. Hence, R < S. 

Then we use Ti-separation property. More precisely, suppose some points pi,...,ps 
belong to Gp. Let us choose some twisted-invariant functionals (pi = (fp- corresponding 
to these points as it was described (i.e. choose some scaling). Assume that = 1, 
<Pi{xi) = 1, Xi G pi{L^{G)). If we can find ai G L^{G) such that (pi{pi{ai)) = (Pi{ai) is 
sufficiently large and Pj{ai), i ^ j, are sufficiently small (in fact it is sufficient Pj{ai) to be 
close enough to Kj := Kp^), then (Pj{ai) are small for i ^ j, and ip* are linear independent 

and hence, s < R. This would imply S := i^Gp < R is finite. Hence, R = S. 

So, the problem is reduced to the search of the above Oj. Let d = max dimpj. For 

i=l,...,s 

each i let Cj := where Xi is the unitary equivalence of pi and (f)pi and Xi = pi{bi). 
Let c := max a and e := ^ 2j ■ 

i=l,...,s 2s^dc 

One can find a positive element a[ G L^{G) such that ||pj(a9|| > 1 and ||pj(ai)|| < £ for 
j 7^ i. Indeed, pi can be separated from one point, and hence from the finite number of 
points: pj, j ^ i. Hence, one can find an element Vi such that ||pj(t'j)|| > 1, ||pj(^^j)|| < 1 
for j ^ i [6, Lemma 3.3.3]. The same is true for the positive element Ui = v*Vi. (Due to 
density we do not distinguish elements of and G*). Now for a sufficiently large n the 
element a\ := (uj)" has the desired properties. 

Let us take Oj := a[h*. Then 



(10) (p*{ai) = Tr{xipi{ai)) = Tr{xipi{a'i)pi{bi)*) = Tr{xipi{a'i)x*) 



TT{x,pi{a'^{x,)-') = Tt{pM)) > > K 

dim Pi a 



For j i one has 



(11) 
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Then the s x s matrix $ = ip*{ai) can be decomposed into the sum of the diagonal matrix 
A and off-diagonal S. By (10) one has A > i. By (11) one has 

MM 2 2 1 1 

S <s ■Ci-e<s -c-- — — — = — . 

2 ■ s'^ ■ a - c 2d 

Hence, $ is non-degenerate and we are done. □ 

Lemma 5.4 together with (8) completes the proof of Theorem 1.4 for automorphisms. 
We need the following additional observations for the proof of Theorem 1.4 for a general 
endomorphism (in which (3) is false for infinite-dimensional representations). 

Lemma 5.5. (1) If (f) is an epimorphism, then G is (j)fi- invariant. 

(2) For any (p the set Iiep{G) \ G is (^^-invariant. 

(3) The dimension of the space of intertwining operators between p E G and 4>Bip) is 
equal to 1 if and only if p E Fix({/)). Otherwise it is 0. 

Proof. (1) and (2): This follows from the characterization of irreducible representation as 
that one for which the centralizer of p{G) consists exactly of scalar operators. 

(3) Let us decompose 0_r(p) into irreducible ones. Since dim Hp = dimH^^^-^ one has 

only 2 possibilities: p does not appear in 0(p) and the intertwining number is 0, otherwise 
(j)R{p) is equivalent to p. In this case p G Fix(</)). □ 

The proof of Theorem 1.4 can be now repeated for the general endomorphism with the 
new definition of Fix(0). The item (3) supplies us with the necessary property. 

6. Examples and their discussion 

The natural candidate for the dual object to be used instead of G in the case when the 
different notions of the dual do not coincide (i.e. for groups more general than type I one 

groups) is the so-called quasi-dual G, i-e. the set of quasi-equivalence classes of factor- 
representations (see, e.g. [6]). This is a usual object when we need a sort of canonical 
decomposition for regular representation or group C*-algebra. More precisely, one needs 

the support Gp of the Plancherel measure. 

Unfortunately the following example shows that this is not the case. 

Example 6.1. Let G be a non-elementary Gromov hyperbolic group. As it was shown 
by Fel'shtyn [11] with the help of geometrical methods, for any automorphism </> of G 
the Reidemeister number -R(0) is infinite. In particular this is true for free group in two 

generators F2. But the support (-F2)p consists of one point (i.e. regular representation is 
factorial). 

The next hope was to exclude from this dual object the //i-points assuming that they 
always give rise to an infinite number of twisted invariant functionals. But this is also 
wrong: 



Example 6.2. (an idea of Fel'shtyn realized in [20]) Let G = {Z (B Z) xig Z he the semi- 

2 1 
1 1 



direct product by a hyperbolic action ^(1) = ( ^ | 1 . Let be an automorphism of G 
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whose restriction to Z is —id and restriction to Z © Z is 







I 1 
1 



) 



Then = 4, 



while the space G-p consists of a single Ili-point once again (cf. [3, p. 94]). 

These examples show that powerful methods of the decomposition theory do not work 
for more general classes of groups. 

On the other hand Example 6.2 disproves the old conjecture of Fel'shtyn and Hill [13] 
who supposed that the Reidemeister numbers of an injective endomorphism for groups 
of exponential growth are always infinite. More precisely, this group is amenable and of 
exponential growth. Also, one has the following example 

Example 6.3. In [32] D. Osin has constructed an infinite finitely generated group, which 
is not amenable, contains the free group in two generators, and has two (ordinary) con- 
jugacy classes. 

The role of this example will be clarified in Section 11. 

In this relation the following example (to be discussed in Section 12) seems to be 
interesting. 

Example 6.4. [12] For amenable and non-amenable Baumslag-Solitar groups Reidemeis- 
ter numbers are always infinite. 

For Example 6.2 recently we have found 4 fixed points of being finite dimensional 
irreducible representations. They give rise to 4 linear independent twisted invariant func- 
tionals. These functionals can also be obtained from the regular factorial representation. 
There also exist fixed points (at least one) that are infinite dimensional irreducible rep- 
resentations. The corresponding functionals are evidently linear dependent with the first 
4. This example will be presented in detail in Section 7. 



In the present section based on [18] it is shown that the twisted Burnside theorem 
(or Fel'shtyn-Hill conjecture) in the original formulation with G as a dual object is not 
true for non-type I groups. More precisely, an example of a group and its automorphism 
is constructed such that the number of fixed irreducible representations is greater than 
the Reidemeister number. But the number of fixed finite-dimensional representations 
(i.e. the number of invariant finite-dimensional characters) in this example coincides with 
the Reidemeister number. The directions for search of an appropriate formulation are 
indicated (another definition of the dual object). Some advances in this direction will be 
made in the next sections. 

Let G be a semidirect product of 1? and Z by Anosov automorphism a with the matrix 



7. An EXAMPLE FOR TYPE IIi GROUPS 




It consists by the definition of triples ((m, A;),n) of integers with the 



following multiplication low: 



((m, k),n) * ((m', k'), n') = ((m, k) + a"-{m', k'),n + n'). 



In particular. 



((m, /c),0)*((0,0),n) = ((m,A;),n). 
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The inverse of {{m,k),n) is {—a~"'{m,k),—n). Indeed, 

((m, k),n)* (-a~"(m, k), -n) = ((m, k) - a"a""(m, k), n - n) = ((0, 0), 0). 

The group G is a solvable (hence, amenable) group which is not of type I. Its regular 
representation is factorial. The irreducible representations can be obtained from ergodic 
orbits of the action of a on the torus which is dual for the normal subgroup using 
appropriate cocycles [3, Sec. II. 4], [2, Ch. 17, § 1]. 

Let us define an automorphism : G — » G by 

0((m, k),n) = ((/c, -m), -n), 

i.e. the action on 7? is defined by automorphism fi with the matrix M = 
and on Z by n —n. The map is clearly a bijection, 

0(((m, k),n)* ((m', k'),n') = 0((m, k) + a"{m', k'),n + n')) 

= (l){{k, —m) + fj,a"'{m' , k'), —n — n), 

(f){{m, k),n) * 0(((m', k'),n') = {{k, — m), —n) * {{k', —m'), —n') 

= {{k, -m) + a~'^{k', -m'), -n - n'). 

Hence, to prove that (j) is an automorphism, we need fia^ = a~'^fi. This follows from 
Ha = a~^fi. The further results in this direction can be found in [20]. 

Let us find the Reidemeister classes of 0, i.e. the classes of the equivalence relation 
h ~ gh(j){g~^). For h = {{m,k),n) and g = {{x,y),z) the right hand side of the relation 
takes the following form: 

(12) ((x, y) + a^{m, k), z + n) * (— /ia~^(x, y), z) 

= ((x, y) + a^{m, k) - a^"^'^/ia"''(x, y),2z + n) 

= (a^{(m, k) + (Id -a»Q;"^(x, y)}, 2z + n). 

Let us call level n (of G) the coset L„ of Z^ C G of all elements of the form ((m, k),n). 
Let us first take an element ((m, /c),0) from the level and describe elements from the 
same level, being equivalent to it. By (12) in this case z = n = and they have the form 

((m, k) + (Id -/i)(x, y),0) = {{m + {x-y),k + {x + y)), 0), 



where Id — has the matrix 




Hence, the level has intersections with 2 



Reidemeister classes, say, Bi and B2. The first intersection Bi HLq is formed by elements 
{{u, v), 0) with even u + v, and i?2 H Lq — with odd u + v. The elements from the other 
levels, which are equivalent to ((m, fc), 0), have the form 

(13) («^{(m, k) + (Id -/i)a"^(x, y)}, 2z). 

This means that Bi and B2 enter only even levels. Also, since a is an automorphism, we 
can rewrite (13) as 

(14) {a'{{m,k) + {ld-fi){u,v)},2z). 

with arbitrary integers u and v. This means, that the intersections Bi{~\L2z have the form 
a^(i?j), i = 1,2. In particular, the other Reidemeister classes do not enter even levels. 
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In a similar way, the elements of Li equivalent to ((m. A;), 1) have the form 

((m, k) + (Id -a/i)(x, y), 1) = ((m + (2x - 2y), k + x), 1). 

This means, that Li enters 2 classes: the intersection with is formed by elements with 
even first coordinate, and with S4 — with the odd one. The elements from the other 
levels, which are equivalent to ((m, /c), 1), have the form 

(15) {a^{{m, k) + (Id -a^i)a~\x, y)}, 2z + l). 
Since a is an automorphism, we can rewrite (15) as 

(16) {a^{{m,k) + (Id -a/i)(u, t;)}, 2z + 1). 

with arbitrary integers u and v. This means, that the intersections Bi fl L2Z+1 have the 
form a^(i?j), z = 3,4. In particular, these four classes cover G and /?(</>) = 4. 

To obtain a complete description of i?j let us remark that directly from the definition 
of a 

aix.y) = {2x + y,x + y) 
one has the following properties: 

• a maps the set of elements with an even (resp., odd) sum of coordinates onto the 
set of elements with an even (resp., odd) second coordinate, 

• a maps the set of elements with an even (resp., odd) second coordinate onto the 
set of elements with an even (resp., odd) first coordinate, 

• a maps the set of elements with an even (resp., odd) first coordinate onto the set 
of elements with an even (resp., odd) sum of coordinates. 

Hence, the elements ((m, k),n) in intersections Bi fl Lj are of the form 



i 


1 


2 


3 


4 


j = mod 6 


m + k is even 


m + A; is odd 








j = 1 mod 6 








m is even 


m is odd 


j = 2 mod 6 


k is even 


k is odd 








j = 3 mod 6 








m + A; is even 


m + k is odd 


j = 4: mod 6 


m is even 


m is odd 








J = 5 mod 6 








k is even 


k is odd 



Now we want to study the fixed points of the homeomorphism (p : G ^ G, [p] ^ 
of the unitary dual. Let us start from the finite-dimensional representations. As it was 
shown in [16] (see also Section 3 and after) in this case there exists exactly one twisted- 
invariant functional on (G) , or 0-central L°° function, coming from a twisted-invariant 
functional on p{L}{G)) = M(dimp, C) (up to scaling), namely 

(17) ^p- 9^ Tr(5p(^)), 

where S is the intertwining operator between p and p0. 

First, we have to find /^-invariant finite a-orbits on T^. One can notice that 

det(v4'' - M) = det A'^ + 1 = 2 

for any n. Hence, the mentioned orbits are formed by points with coordinates and 1/2. 
We have 2 orbits: one of them consists of 1 point (0, 0) and gives rise to 1-dimensional 
trivial representation pi, and the other consists of Ai = (0,1/2), A2 = (1/2,0) and 
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A3 = (1/2,1/2) and gives rise to a 3- dimensional (irreducible) representation p2- Also, 
one has the following 1-dimensional representation vr: 

7i{{m,k),2n) = 1, 7i{{m, k),2n + 1) = -1. 

So, we have 4 representations 

Pi, P2, TT, P2®vr. 

We claim that they give rise via (17) to 4 linear independent twisted- invariant functionals. 
In particular, there is no more finite-dimensional fixed points of (j). 
Clearly, 

1, on Un L2n = BiU B2, 
-1, on Un L2n+1 = B3 U B4^, 



(18) 



1, 



Let us find ipp^. In the space L'^{Ai, A2, A3) we take the base ei, 62, £3 of characteristic 
functions in these points. One has 

a{ei) = £3, a{e3) = 62, a{e2) = ei, 

At(£^i) = £2, ^£2) = £1, P'iss) = £3- 
The representation (see [2, Ch. 17, § 1]) is defined by: 

P2{m,k,0){ei) = xaA^^^) ■ Si, p2{0,0,n){ei) = a'^'isi) = Si+n mods, 
P2(m,A;,0)(£i)=e2-(°""+i/2-^) 
P2(m,A;,0)(e2)=e2'^^(^/2-™+°-^-) 
P2{m, k, 0)(e3) = e^-^(^/2-^-V2-^) . 53 = e--- ■ -.3 
The representation (f)p2 is defined by 

0p2(O,O,n)(£:i) = p2(0,0, -n)(5i) = mods, 
0P2(m, A;, 0)(£i) = p2(A;, -m, 0)(ei) = e'^^'^ei 

^P2(m, /c, 0)(e2) = P2{k, -m, 0)(e2) = e'"'''e2, 

#2(m. A;, 0)(£3) = P2ik, -m, 0)(£3) = e-^(^-™)£3 = e^'^^^^^Ss 

^010 

The intertwining operator is induced by and has the matrix S 











= e^'^ei. 


^■82 




= e'^'"^£2, 


■£3 = 


7vi{m-\-k) 

— 



1 
1 



Hence, 



Tr 



(fp^{m,k,n) = Tr{Sp2{m,k,0)p2{0,0,n)) 
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For n = mod 3 

iPp^im, k,n) = Tr 
for n = 








mod 3 

fp^im, k,n) 








Tr 



for n 







-3 7ri(A;+-m) 












^7rj(fc+m) 



1, if m + is even 
-1, a m + k is odd 









if m is even 
if m is odd 




















1, 
-1, 



if k is even 
if k is odd 



(/)p2 is 3-periodical in n, while the characteristic functions of Bi are 6-periodical. For 
j = 0, . . . , 5 one has 



^P2 


BiHLo 


= 1, 


fp2 


B2nLo = 


-1, 


^P2 




= 1, ^P2 


BiHLi 


-1, 




BinL2 


= 1, 


^P2 


B2nL2 = 


-1, 


¥>P2 


B^nLs 


= 1, ¥>P2 


- = 


-1, 


VP2 


BinLi 


= 1, 


'fp2 


B2nLi = 


-1, 


fp2 


BsCiLs 


= 1, ^P2 


S4nL5 = 


-1, 



so ^JpalsiuBs = 1, V5p2ls2UB4 = —1- The determinant of the values of the functions (pp^^, 
V-iT, </3p2) Vp2<s>TT on Bi, B2, B3, B4 is 



det 



/ 1 

1 
1 



1 \ 



-8 ^ 0. 



Hence, they are linearly independent. 

Nevertheless, there are infinite-dimensional irreducible ^-invariant representations. E.g. 
we have a representation p of G on L^(T^) with the respect to the Lebesgue measure, with 
p{m, k, 0) be the multiplier by characters in the appropriate points and p(0, 0, 1) is a, in 
the same manner as for p2. 

This disproves the conjecture of Fel'shtyn and Hill [13], who supposed that the Reide- 
meister number equals to the number of fixed points of (p on G. 

This representation is not traceable, but one can nevertheless try to calculate (17). Let 
us choose an orthonormal base of L^(T^) formed by est{x,y) 
The intertwining operator S is generated by /i. Then 



^2ni{sx+ty)^ X, ?/ G [0,1]. 



Tr(S'p(m, k, n)) 



s,t 



^0 



(p(m, k, 0)p(0, 0, n)£:,,t)(p(x, y))es,tix, y) dx dy 
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^{MM^,y)) (^(0, 0, n)es,t) {fi{x, y))e-s,-t{x, y) dx dy 




,2iTi{{m,k),fi[x,y)) 



y))e-s-t{x, y) dx dy 




,2ni{{'m,k)+{a' 



E 1- 



For n = this equals 1 for m + k even and for m + k odd. Hence, ^plsinLo = 1? 
<Pp\B2nLo = 0. For n = 1 the equahty takes the form {m,k) = (—2s — 2t,—t). Hence, 
<^p|B3nLi = 1, V^p|B4nLi =0. If n = 2r, the equahty takes form 



Since a is an automorphism of Z © Z and by the description of Bi via the action of a, we 
obtain that (/^pl^^ = 1, y^pl^j = 0. Similarly, for n odd. So, ipp is well defined and 



Of course, there are also {/)-invariant traceable factor representations of this group G, 
e.g. the regular representation. Since its kernel is trivial, evidently all twisted-invariant 
functionals can be pushed back from it. 

This example shows that the conjecture of [13] for general groups can survive only after 
eliminating badly separated points in G. 



Let us first remind the theory of operator fields following [9]. Let T be a topological 
space and for each point t E T a, C*-algebra (or more general — involutive Banach algebra) 



Definition 8.1. A continuity structure for T and the {At} is a linear space F of operator 
fields on T, with values in the {A^}, (i.e. maps sending t G T to an element of At), 
satisfying 

(1) if X G -F, the real-valued function t \\x{t)\\ is continuous on T; 

(2) for each t E T, {x{t) \ x G F} is dense in Af, 

(3) F is closed under pointwise multiplication and involution. 

Definition 8.2. An operator field x is continuous with respect to F at to, if for each 
e > 0, there is an element y & F and a neighborhood U of to such that \\x{t) — y(t)|| < e 
for all t eU . The field x is continuous on T if it is continuous at all points of T. 

Definition 8.3. A full algebra of operator fields is a family A of operator fields on T 
satisfying: 

(1) A is a *-algebra, i.e., it is closed under all the pointwise algebraic operations; 

(2) for each x E A, the function t i— s> ||a;(t)|| is continuous on T and vanishes at infinity; 

(3) for each t, {x{t) \ x E A} is dense in At; 



(m, k) = {ji — a ^)(s, t), (m, k) = {a^fia^ — a ^)(s, t), 
a~^{m, k) = (jj — a^)a^'{s, t). 



8. Algebras of operator fields 



At is fixed. 
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(4) A is complete in the norm = sup ||a;(if:)||. 



A full algebra of operator fields is evidently a continuity structure. If F is any continuity 
structure, let us define Co{F) to be the family of all operator fields x which are continuous 
on T with respect to F, and for which t i— > ||a;|| vanishes at infinity. One can prove that 
Co{F) is a full algebra of operator fields — indeed, a maximal one. 

Lemma 8.4. For any full algebra A of operator fields on T, the following three conditions 
are equivalent: 

(1) A is a maximal full algebra of operator fields; 

(2) A = Cq{F) for some continuity structure F; 



Such a maximal full algebra A of operator fields may sometimes be called a continuous 
direct sum of the {At}. It is clearly separating, in the sense that , if s, t G T, s 7^ t, 
a e As, (3 G At, there is an x G A such that x{s) = a, x(t) = (3. We will denote by a the 
section corresponding to an element a ^ A. We will study the unital case and T will be 
compact without the property of vanishing at infinity. The corresponding algebra will be 
denoted by r(^) = A. Moreover, we suppose that T is Hausdorff, hence, normal. 



Definition 9.1. A [bounded additive) algebra of operator field measure (BA AOFM) 
related to a maximal full algebra of operator fields A = T{A) is a set function fi : S —* 
r(^)* = A*, where 5* G S, some algebra of sets, 

• being additive: yu(US'j)(a) = ^j/^(5'j)(a) 

• fi{S){a) = if supp a n S" = 0. 

• bounded: sup over partitions {Si} of T of J2i \\f^{^i)\\ is finite and denoted by 

Definition 9.2. It is *-weak regular (RBA AOFM) if for each E e T., a e A, and e > 
there is a set F G S whose closure is contained in E and a set G G S whose interior 
contains E such that ||/i(C)a|| < e for every C G S with C G G\F. 

We will use as S all sets and the algebra generated by closed sets in T. 

Definition 9.3. Let AOFM A be defined on an algebra E of sets in T and A(0) = 0. A 
set G S is called X-set if for any M G S 



Lemma 9.4. Let A be an AOFM defined on an algebra S of sets in T with A(0) = 0. The 
family of \- sets is a subalgebra ofT, on which A is additive. Furthemore, if E is the union 
of a finite sequence {En} of disjoint X-sets and M G S, then X{Mr\E) = X{M (1 En). 

Proof. It is clear, that the void set, the whole space, and the complement of any A-set are 
A-sets. Now let X and Y be A-sets, and M G S. Then, since X is a A-set, 



(3) A = Co{A). 



9. FUNCTIONALS AND MEASURES 



A(M) = A(M n + A(M n (T \ E)). 




A(M nY) = x{M nYnx) + a(m n f n (t \ x)) 



A(M) = A(M n F) + A(M n (T \ Y)), 
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A(M n (T \ (X n Y))) = A(M n (T \ (X n Y)) n F) + A(M n (T \ (X n Y)) n (T \ Y)), 

hence, 

(21) A(M n (T \ (X n F))) = A(M n (T \ X) n F) + A(M n (T \ Y)). 

From (19) and (20) it follows that 

A(M) = A(M n F n X) + A(Af n F n (T \ X)) + A(M n (T \ F)), 
and from (21) that 

A(M) = A(M n F n X) + X{M n (T \ (X n F))). 

Thus X n F is a A-set. Since UX„ = T \ n(T \ X„), we conclude that the A-sets form an 
algebra. Now if Ei, and E2 are disjoint A-sets, it follows, by replacing M by Mfl (-Ei U-E'2) 
in Definition 9.3, that 

A(M n (^1 u ^2)) = A(M n ^1) + A(M n E2). 

The final conclusion of the lemma follows from this by induction. □ 

As it is well known, any functional r on a C*-algebra B can be represented as a linear 
combination of 4 positive functionals in the following canonical way. First let us represent 
r under the form t = ti + iT2, where self-adjoint functionals ti and T2 are defined by the 
formulas 



(22) n(a) = r2(a) - - 



2 ' ' ' 2i 

By the lemma about Jordan decomposition, any self-adjoint functional a can be repre- 
sented in a unique way as a difference of two positive functionals a = — under 
requirement 

(23) ||a|| = + 

(see [29, §3.3], [33, Theorem 3.2.5]). Let us decompose an AOFM in the related way. 
Since the decomposition is unique, the second property of AOFM will held. If we start 
from BA AOFM, then the additivity of summands will follow from the uniqueness of the 
decomposition, and the boundedness (with double constant) will follow from (22) and 
property (23). The same argument shows that the summands are *-weak regular, if the 
initial AOFM was *-weak regular. So, the AOFM's in the decomposition are positive 
AOFM, i.e., such that 

fx{E)[a*a] > 
for any E & T,. Such a set function is non-decreasing. 

Lemma 9.5. The sets F and G in the Definition 9.2 can be chosen in such a way that 
\\fi{C){fa)\\ < e for any continuous function / : T [0, 1]. 

Proof. Let us take the decompositions /i = J2t=i ^if^iy ^ = Si=i where /ij and aj are 
positive, Xi, yj are complex numbers of norm < 1. Let us choose the sets F and G, as in 
Definition 9.2, for e/16 and for all pairs /ij, aj simultaneously. Then 

< /i.(C)(/ ■ a,) = ii,{G){{a,f"f{a,f") < fi.{G){a,) < ^, 
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and 

4 

e 



^l^{C){f ■a)\\< \^^y^ ■ \^^^{C){a,)\ < le ■ - = e. 

□ 



Theorem 9.6. Let a unital separable C*-algebra A be isomorphic to a full algebra of 
operator fields T{A) over a Hausdorff space X . Then functionals on A = T{A) can be 
identified with RBA AOFM of A. 

Proof. Let us remark that these suppositions imply the foUowing: T is a separable Haus- 
dorff compact and the unit ball of the dual space of A is a metrizable compact in *-weak 
topology. 

Obviously RBA AOFM form a linear normed space with respect to ||.||. 

First, we want to prove that the natural linear map fi ^ 1^{T) is an isometry of RBA 
AOFM into A* . Since ||/i(T)|| < it is of norm < 1. Let now take an arbitrary small 
£ > 0. Let . . . , E'n be a partition of T such that 

n 
i=l 

Let tti E Ahe such elements of norm 1, that fi{Ei){ai) > \\fi{Ei)\\ — e/n. 

By *-weak regularity of and normality of T one can take closed sets Ci, disjoint open 
sets Gi, and continuous functions fi'.T-^ [0, 1] such that Ct C \ Cj)(aj) || < 

s/n'^, Ci C Gi, \\fi{Gi \ Gi){aj)\\ < e/n"^, (and estimations hold for multiplication by 
positive functions as well, as in Lemma 9.5) = if s ^ Gi, fi{s) = 1 if s G Cj, 

i,j = l,...,n. 

Consider the element a := fiQi G r(^) = A. Then ||a|| < 1 and 

n 

I /i(5)(a) - ||/i|| I < HE,){a) - fi{Ei)\ + e 

i=l 

n 

< Y \l^{E, \ G,){a) + ^i{Ci){a) - /i(^.)(a,)| + 2e 



i=l 



i=l 



+ 2e 



n 

e e 



i,j=l i=l 

Since e is arbitrary small, = ||/i(S')||. 

It remains to represent the general functional if by RBA AOFM. This functional on 
r(.4) can be extended by Hahn-Banach theorem to a continuous functional tp on B{A) = 
Ylt^T^t (^^^ C*-algebra of possibly discontinuous cross-sections of A with sup- norm). 
This functional can be decomposed ip = Ylt=i<^i'^ij where ipi are positive functionals, 
ai G C, \ai\ = 1, \\ipi\\ < \\ip\\- Let us define 

X{E){a) := tlj{xEa), Xi{a) := ipi{xEa), i = l,...,A. 
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where a G r(^), and xe is the characteristic function of E. Evidently, A(T)(a) = il){a) 
and A is a BA AOFM. Indeed, the first two properties of Definition 9.1 are evident. The 
third one can be verified for each Aj, z = 1, . . . , 4: 

N N N 

j=i j=i j=i 

hence, 

N AN 4 

j=i 1=1 j=i 1=1 

Now we want to find an RBA AOFM /i such that fi(T){a) = A(T)(a). Without loss of 
generality it is sufficient to do this for a positive A = Aj. 

Let F represent the general closed subset, G the general open subset, E the general 
subset of S. Define /ii and fi2 by putting 

/ii(F)(a*a) = inf X{G){a*a), fi2{E){a*a) = sup yUi(F)(a*a), 

gdf P(-e 

and then by taking linear extension. More precisely, due to separability one can take a 
cofinal sequence {Gj}. The unit ball in the dual space is weakly compact and one can take 
a weakly convergent sequence \{Gi^). Its limit is a, positive functional on A enjoying 
inf-property on positive elements. In particular, it is independent of the choice of {Gi} 
and {Gjj.}. In a similar way for sup. 

These set functions are non-negative and non- decreasing. Let Gi be open and Fi be 
closed. Then if G D Fi \ d, it follows that d n G D Fi and A(Gi) < A(Gi) + A(G) so 
that /ii(-Fi) < A(Gi) + A(G). Since G is an arbitrary open set containing Fi \ Gi we have 

/ii(i^i)<A(Gi)+/ii(Fi\Gi). 

If F is a closed set it follows from this inequality, by allowing Gi to range over all open 
sets containing F fl Fi, that 

/^i(i^i) < yUi(F n Fi) + ^2(Fi \ F). 

If E is an arbitrary subset of T and Fi ranges over the closed subsets of F, then it follows 
from the preceding inequality that 

(24) /i2(F) < ii2{E n F) + ^2(^ \ F). 

It will next be shown that for an arbitrary set F in T and arbitrary closed set F in T we 
have 

(25) /i2(F) > ii2{E n F) + ii2{E \ F). 

To see this let Fi and F2 be disjoint closed sets. Since T is normal there are disjoint 
neighborhoods Gi and G2 of Fi and F2 respectively. If G is an arbitrary neighborhood of 
Fi U F2 then A(G) > A(G n Gi) + A(G n G2) so that 

/il(FinF2) >/ii(Fi) +/i2(F2). 

We now let F and F be arbitrary sets in T with F closed and let Fi range over closed 
subsets of FnF while F2 ranges over the closed subsets of F\F. The preceding inequality 
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then proves (25). From (24) and (25) we have 

(26) fi2{E) = fi2{E n F) + /i2(F n (T \ F)) 

for any F in T and F closed. The function fi2 is defined on the algebra of all subsets of T 
and it follows from (26) that every closed set F is a /i2-set. If /i is the restriction of /i2 on 
the algebra determined by the closed sets, it follows from Lemma 9.4 that /i is additive 
on this algebra. It is clear from the definition of fii and /i2 that yUi(F) = /i2(F) = yu(F) if 
F is closed and thus fi{E) = supp^-^ f^{E)- This shows that fi is *-weak regular and since 
||/i(T)|| < oo, we have /i is RBA AOFM. 
Finally, by the definition, fi{S){a) = \{S){a) = ip{a) = ip{a) for a G T{A). □ 

10. Twisted-invariant AOFM 

The most part of the argument is valid for various representations of algebras by op- 
erator fields, but now we will pass to the case of group C*-algebra of a discrete group 
and concentrate ourselves on the following important case due to Dauns and Hofmann [4, 
Corollary 8.13] (see also [4, Corollary 8.14]): 

Theorem 10.1. Consider a C* -algebra A [with or without an identity) and the set B of 

all its primitive ideals in the hull-kernel topology q. 

(1) The complete regularization (f : {B,q) iM,t) can be taken to consist of closed 
ideals m of A satisfying m = r\(p~^{m). The topology t of M contains the weak-star 
topology t* induced on M by A. 

Denote by K, the family of all t- compact subsets of M of the form {m G M|||a+m|| > e}, 
a G A, < e. Let n" : E' {M,t) be the uniform field of C*- algebras obtained by first 
forming the canonical field from A, M and then enlarging the weak-star topology t* of M 
up to t. For each a ^ A, a is the map a : M — > E' , d{m) = a-|-m G A/m. The C* -algebra 
of all sections vanishing at infinity with respect to the class K, is denoted by ro(7r"). 

(2) Then the map A — s> ro(vr"), a ^ a is an isometric star-isomorphism. In particular, 
if M is t-compact, then M G /C and A = r(7r"). 

Definition 10.2. Ideals m from the previous theorem, i.e., points of M, are called Glimm 
ideals. 

We will use the notation T instead of M for Glimm spectrum to agree with the previous 
section. 

Now we consider a countable discrete group G and its automorphism 0. Also, A = 
G*{G). One has the twisted action of G on A: 

g[a] = LgaL^(_g-i), 

where Lg is the left shift, and respectively on functionals. 

Definition 10.3. The dimension R^{(f)) of the space of twisted invariant functionals on 
C*{G) is called generalized Reidemeister number of (f). 

Definition 10.4. A (Glimm) ideal J is a generalized fixed point of (j) if the linear span of 
elements h — g\b] is not dense in Ai = A/ 1, where g[.] is the twisted action, i.e. its closure 
Ki does not coincide with Aj. 
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If we have only a finite number of such fixed points, then the twisted invariant RBA 
AOFM's are concentrated in these points. Indeed, let us describe the action of G on RBA 
AOFM's in more detail. The action of G on measures is defined by the identification of 
the measures and functionals on A. 

Lemma 10.5. // /i corresponds to a twisted invariant functional, then for any Borel 
E G T the functional fi{E) is twisted invariant. 

Proof. This is an immediate consequence of *-weak regularity. Let a G A, g G G and 
£ > be an arbitrary small number, and U and F as in Lemma 9.5, for a and g[a] 
simultaneously. Let us choose a continuous function / : T — > [0, 1], with f\p = 1 and 
f\T\u = 0. Then 

|/i(E)(a - g[a])\ = \ F)(a) + /i(F)(a) - \ F)ig[a]) - fiiF)ig[a])\ 

< |/i(F)(a) - /i(F)((7[a])| + 2e = HF){fa) - K^MH)] + 2e 
= \fi{U){fa) - ^(f/ \ F)(/a) - fiiUMfa]) + ^^{U \ F){g[fa])\ + 2e 
< \fi{U){fa) - KUMfa])\ + 45 = |/i(T)(/a) - ^(T)(^?[/a])| + As = As. 

□ 

Lemma 10.6. For any twisted-invariant functional if on G*{G) the corresponding mea- 
sure fi is concentrated in the set GFP of generalized fixed points. 

Proof. Let ||/i|| = 1. Suppose opposite: there exists an element a E A, \\a\\ = 1, vanishing 
on generalized fixed points, such that ip{a) ^ 0. Let e := |v5(a)| > 0. In each point 
t ^ GFP we can find elements h\G A^ g\ G G^ i = \^ . . . k{t), such that 

m 

Ht)-J2(9i[bm-m)\\<e/A. 

i=l 

Then there exists a neighborhood Ut such that for s G Ut one has 

kit) 

\\a{s)-J2i9ims)-b]ism<e/2. 

i=l 

Let us choose a finite subcovering {Ut^}, j = l,...,n, of {Ut} and a Borel partition 
El, . . . , En subordinated to this subcovering. Then 

n n / k{tj) \ „ k{tj) 

^{a) = J2fi{E,){a) = J2fi{E,) a - J^igl^H^] - b]^) + f^i^MK^ " 

j=i j=i \ i=i J j=i i=i 

By Lemma 10.5 each summand in the second term is zero. The absolute value of the first 
term is less then J2j < ||/^|| ■ = e/2. A contradiction with \^{a)\ = e. □ 

Since a functional <^ on Aj is twisted-invariant if and only if Ker D Kj, the dimension 
of the space of these functionals equals the dimension of the space of functionals on Ai / Kj 
and is finite if and only if the space Aj / Kj is finite dimensional. In this case the dimension 
of the space of twisted-invariant functionals on Aj equals dim{Aj/Kj). 
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Definition 10.7. Generalized number S'*(0) of fixed points of on Glimm spectrum is 

leGFP 

Since functionals associated witli measures, whicli are concentrated in different points, 
are linearly independent (the space is Hausdorff), the argument above gives the following 
statement. 

Theorem 10.8 (Weak generalized Burnside). 

if one of these numbers is finite. 

In [16] (see also [10, 18] and the first part of the present survey paper) it was proved 
for groups of type I that the Reidemeister number R{(f)), i.e., the number of twisted 
(or 0-)conjugacy classes, coincides with the number of fixed points of the corresponding 
action on the dual space G. This was a generalization of Burnside theorem. The proof 
used identification of R{4>) and the dimension of the space of {L°°-) twisted invariant 
functions on G, i.e. twisted invariant functionals on L^{G). Since only part of 
functions defines functionals on C*{G) (namely, Fourier-Stieltjes functions), so, a priori 
one has R^:{(j)) < R{<f))- Nevertheless, the functions with some symmetry conditions 
very often are in Fourier-Stieltjes algebra, so one can conjecture that R{(f)) = R*{4>) if 
R{4>) < oo. This is the case for all known examples. 

11. Finite-dimensional representations and almost polycyclic groups 

Let us start from several useful facts. 

Theorem 11.1 ([34, Theorem 1.41]). If G is a finitely generated group and H is a 
subgroup with finite index in G, then H is finitely generated. 

Lemma 11.2. Let G be finitely generated, and H' G G its subgroup of finite index. Then 
there is a characteristic subgroup H G G of finite index, H G H'. 

Proof. Since G is finitely generated, there is only finitely many subgroups of the same 
index as H' (see [23], [27, § 38]). Let H be their intersection. Then H is characteristic, 
in particular normal, and of finite index. □ 

Theorem 11.3 (see [25]). Let A be a finitely generated Abelian group, ip : A ^ A its 
automorphism. Then R{ip) = # Coker('0 — Id), i.e. to the index of subgroup generated by 
elements of the form x~^ip{x) . 

Proof. By Lemma 2.1, Ri^ip) is equal to the index of the subgroup H = {e}^. This group 
consists by definition of elements of the form x~^ip{x). □ 

Let us remind the following definitions of a class of groups. 

Definition 11.4. A group with finite conjugacy classes is called FC-group. 

In a FC-group the elements of finite order form a characteristic subgroup with locally 
infinite abelian factor group; a finitely generated FC-group contains in its center a free 
abelian group of finite index in the whole group [30]. 
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Lemma 11.5. An automorphism (p of a finitely generated FC-group G with R{4>) < oo 
has a finite number of fixed points. 

The same is true for o (p. Hence, the number of g E G such that for some x G G 

gx(t){g~^) = X, 

is still finite. 

Proof. Let A be the center of G. As it was indicated, A has a finite index in G and hence, 
by Theorem ILl, is f.g. Since A is characteristic, one has an extension A G G/A 
respecting 0. One has < ■ \G/A\ by Lemma 4.3 and #Fix(0') < R{^) ■ \G/A\ 

by Theorem 4.5. Then #Fix(0) < ■ IG/Ap by Lemma 4.4. □ 

Definition 11.6. We say that a group G has the property RP if for any automorphism 
(j) with R{(f)) < oo the characteristic functions / of Reidemeister classes (hence all 
0-central functions) are periodic in the following sense. 

There exists a finite group W, its automorphism (piy, and epimorphism F : G ^ W 
such that 

(1) The diagram 

</> 

G 

(j>W 

commutes. 

(2) / = F*fw, where fw is a characteristic function of a subset of W. 

If this property holds for a concrete automorphism (j), we will denote this by RP(0). 

Remark 11.7. By (2) there is only one class {g}4, which maps onto {F{g)}(i,^^. Hence, 
F induces a bijection of Reidemeister classes. 

Lemma 11.8. Suppose, G is f.g. and R{(f)) < oo. Then characteristic functions of (p- 
conjugacy classes are periodic {i.e. G has RP(0) ) if and only if their left shifts generate 
a finite dimensional space. 

Proof. From the supposition of finite dimension it follows that the stabilizer of each 0- 
conjugacy class has finite index. Hence, the common stabilizer of all 0-conjugacy classes 
under left shifts is an intersection of finitely many subgroups, each of finite index. Hence, 
its index is finite. By Lemma 1L2 there is some smaller subgroup Gs of finite index 
which is normal and 0-invariant. Then one can take W = G/Gs- Indeed, it is sufficient 
to verify that the projection F is one to one on classes. In other words, that each coset 
of Gs enters only one 0-conjugacy class, or any two elements of coset are 0-conjugated. 
Consider g and kg, g E G, h E Gg- Since h by definition preserves classes, kg = xg(f){x~^) 
for some x E G, as desired. 

Conversely, if G has RP(0), the class {g}^ is a full pre-image F~^{S) of some class 
S G W. Then its left shift can be described as 

9'{9h = g'F~\S) = {g'g^\g^eF-\S)} = {g\{gr'9^F-\S)} 

= {gmg'r'g) e S} = {g\Fig) e Fig')iS)} = F-\Fig')iS)). 

Since W is finite, the number of these sets is finite. □ 
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Remark 11.9. 1) In this situation in accordance with Lemma 11.2 the subgroup Gs is 
characteristic, i.e. invariant under any automorphism. 

2) Also, the group G/ Gs will serve as W (i.e. give rise a bijection on sets of Reidemeister 
classes) not only for (j) but for TgO (p for any g E G because they have the same collection 
of left shifts of Reidemeister classes by Lemma 4.1. 



1) satisfies the following conditions: 



Theorem 11.10. Suppose, the extension 

(1) H has RP; 

(2) G/H zs FC fig. 

Then G has RP(0). 

Proof. We have R{(f)) < oo, hence #Fix(0) < oo by Lemma 11.5 as well as ^Fix{Tz(j)) < 
oo for any z G G/H. Then by Lemma 4.3 R{Tg(f)') < oo for any g E G. Let gi,...,gs, 
s = R{4>), be elements of G which are mapped by p to different Reidemeister classes of 
(j). Now we can apply the supposition that H has RP and find a characteristic subgroup 
Hw := KerF G H oi finite index such that F : H ^ W gives rise a bijection for 
Reidemeister classes of each r^. o0', i = 1, . . . , s, moreover, it is contained in the stabilizer 
of each twisted conjugacy class of each of these automorphisms. We choose it in a way to 
satisfy Remark 11.9. In particular, it is normal in G. Hence, we can take a quotient by 
Hy/ of the extension (1): 




G/H 



H/Hw^ 



The quotient map Fi : G ^ G/ Hw takes {g}^ to {g}ct, and it is a unique class with this 
property (we conserve the notations e, g., (p for the quotient objects). Indeed, suppose two 
classes are mapped onto one. Hence, gih and gihhw belong to the same (different) classes 
as g and gh-^. Moreover, they can not be 0-conjugate by elements of H. Hence (cf. (4)), 
the elements h and hhw are not (r^^ o 0')-conjugate in H . But this contradicts the choice 
of Hw 9 hw- 

By Lemma 11.8 (applied to Gi and concrete automorphism 0) for the purpose to find 
a map F2 : Gi ^ Wi with properties (1) and (2) of the Definition 11.6 it is sufficient to 
verify that shifts of the characteristic function of {h}^ C Gi form a finite dimensional 
space, i.e. the shifts of {h}^ C Gi form a finite collection of subsets of Gi. After that one 
can take the composition 

to complete the proof of theorem. 

Let us observe, that we can apply Lemma 11.8 because the group Gi is finitely gener- 
ated: we can take as generators all elements of W and some pre- images s{zi) e Gi under 
p of a finite system of generators Zi for P. Indeed, for any x G Gi one can find some 
product of Zi to be equal to p{x). Then the same product of s{zi) differs from x by an 
element of W . 
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Let us prove that the mentioned space of shifts is finite-dimensionaL By Lemma 4.1 
these shifts of {h}^ C Gi form a subcollection of 

{x}ryo<p, x,yeGi. 

Hence, by Corollary 4.2 it is sufficient to verify that the number of different automorphisms 
Ty : Gi ^ Gi is finite. 

Let Xi, . . . ,Xn be some generators of Gi. Then the number of different Ty does not 
exceed 

n n 

n \yeG-,}<l[\W\- #{r,(p(x,)) | z e T}, 

where the last numbers are finite by the definition of FC for F. □ 

Now we describe using Theorem 11.10 some classes of RP groups. Of course these 
classes are only a small part of possible corollaries of this theorem. 

Let G' = [G, G] be the commutator subgroup or derived group of G, i.e. the subgroup 
generated by commutators. G' is invariant under any homomorphism, in particular it is 
normal. It is the smallest normal subgroup of G with an abelian factor group. Denoting 
:= G, G'(^) := G', G^""^ := (G^^^-^))', n > 2, one obtains derived series of G: 

(27) G = G(°) dG' D G(2) D . . . D G(") D ... 

If G^^'' = e for some n, i.e. the series (27) stabilizes by trivial group, the group G is 
solvable] 

Definition 11.11. A solvable group with derived series with cyclic factors is called poly- 
cyclic group. 

Theorem 11.12. Any poly cyclic group is RP. 

Proof. By Lemma 2.1 any commutative group is RP. Any extension with H being the 
commutator subgroup G' of G respects any automorphism of G, because G' is evidently 
characteristic. The factor group is abelian, in particular FC. 

Since any polycyclic group is a result of finitely many such extensions with finitely gen- 
erated (cyclic) factor groups, starting from Abelian group, applying inductively Theorem 
11.10 we obtain the result. □ 

Theorem 11.13. Any finitely generated nilpotent group is RP. 

Proof. These groups are supersolvable, hence, polycyclic [35, 5.4.6, 5.4.12]. □ 

Theorem 11.14. Any finitely generated group of polynomial growth is RP. 

Proof. By [21] a finitely generated group of polynomial growth is just as finite extension 
of a f.g. nilpotent group H . The subgroup H can be supposed to be characteristic, i.e. 
(t){H) = H for any automorphism (p : G ^ G. Indeed, let if' C G be a nilpotent subgroup 
of index j. Let H be the subgroup from Lemma 11.2. By Theorem 11.1 it is finitely 
generated. Also, it is nilpotent as a subgroup of nilpotent group (see [27, § 26]). 

Since a finite group is a particular case of FC group and f.g. nilpotent group has RP 
by Theorem 11.13, we can apply Theorem 11.10 to complete the proof. □ 

The proof of the following twisted Burnside theorem can be extracted from Theorem 
11.18, but we formulate it separately due to importance of the classes under consideration. 
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Theorem 11.15. The Reidemeister number of any automorphism (f) of a f.g. group of 
polynomial growth or poly cyclic group is equal to the number of finite- dimensional fixed 
points of (j) on the unitary dual of this group if R{4>) is finite. 

Theorem 11.16. Any almost polycyclic group, i.e. an extension of a polycyclic group 
with a finite factor group, is RP. 

Proof. The proof repeats almost literally the proof of Theorem 11.14. One has to use the 
fact that a subgroup of a polycyclic group is polycyclic [35, p. 147]. □ 

Definition 11.17. Denote by Gf the subset of the unitary dual G related to finite-di- 
mensional representations. 

Theorem 11.18 (Twisted Burnside Theorem). Let G be an RP group and its auto- 
morphism. Denote by Sf{(f)) the number of fixed points of (pf on Gf. Then 

if the Reidemeister number R{(f)) is finite. 

Proof. Let us start from the following observation. Let S be the universal compact group 
associated with G and a : G — S the canonical morphism (see, e.g. [6, Sect. 16.1]). Then 
G/ = S [6, 16.1.3]. The coefficients of (finite-dimensional) non-equivalent irreducible 
representations of S are linear independent by Peter- Weyl theorem as functions on S. 
Hence the coefficients of finite-dimensional non-equivalent irreducible representations of 
G as functions on G are linearly independent as well. 
It is sufficient to verify the following three statements: 

1) If R{(f)) < oo, than each 0-class function is a finite linear combination of twisted- 
invariant functionals being coefficients of points of Fix0j. 

2) If p G Fix 0j, there exists one and only one (up to scaling) twisted invariant functional 
on p{G*{G)) (this is a finite full matrix algebra). 

3) For different p the corresponding 0-class functions are linearly independent. This 
follows from the remark at the beginning of the proof. 

Let us remark that the property RP implies in particular that {/)-central functions (for </> 
with R{(f)) < oo) are functionals on C*{G), not only L^{G), i.e. are in the Fourier-Stieltijes 
algebra B{G). 

The statement 1) follows from the RP property. Indeed, this 0-class function / is a 
linear combination of functionals coming from some finite collection {p,/} of elements of 
Gf (these representations pi, . . . , p^ are in fact representations of the form vTj oF, where vTj 
are irreducible representations of the finite group W and F : G ^ W, as in the definition 
of RP). So, 

s 

/ = 5^/.°P., p. :G^End(y,), /. : End(y,) ^ C, p,^p^,{t^j). 

i=l 

For any g,'g & G one has 

s s 

Y.fM9)) = fig) = figMg-')) = Y.f^^P^^9Mg~')))- 

i=l i=l 
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By the observation at the beginning of the proof concerning hnear independence, 

fiipiig)) = MpiiggHg^^))), i = i, . . . , s, 

i.e. fi are twisted-invariant. For any p G G/, p : G ^ End(y), any functional uj : 
End(y) — >■ C has the form a i— > Tr(6a) for some fixed h G End(V). Twisted invariance 
imphes twisted invariance of h (evident details can be found in [16, Sect. 3]). Hence, 
h is intertwining between p and p o and p G Fix(0j). The uniqueness of intertwining 
operator (up to scaling) implies 2). □ 

Remark 11.19. We were able to prove only one statement of the theorem in the terms 
of S because of difficulties with an extension of to S. 

Now let us present some counterexamples to this statement for pathological (monster) 
discrete groups. Suppose, an infinite discrete group G has a finite number of conjugacy 
classes. Such examples can be found in [36] (HNN-group), [31, p. 471] (Ivanov group), 
and [32] (Osin group). Then evidently, the characteristic function of unity element is not 
almost-periodic and the argument above is not valid. Moreover, let us show, that these 
groups give rise counterexamples to above theorem. 

Example 11.20. For the Osin group the Reidemeister number i?(Id) = 2, while there 
is only trivial (1-dimensional) finite-dimensional representation. Indeed, Osin group is 
an infinite finitely generated group G with exactly two conjugacy classes. All nontrivial 
elements of this group G are conjugate. So, the group G is simple, i.e. G has no nontrivial 
normal subgroup. This implies that group G is not residually finite (by definition of 
residually finite group). Hence, it is not linear (by Mal'cev theorem [28], [35, 15.1.6]) and 
has no finite-dimensional irreducible unitary representations with trivial kernel. Hence, 
by simplicity of G, it has no finite-dimensional irreducible unitary representation with 
nontrivial kernel, except of the trivial one. 

Let us remark that Osin group is non-amenable, contains the free group in two gener- 
ators -F2, and has exponential growth. 

Example 11.21. For large enough prime numbers the first examples of finitely gener- 
ated infinite periodic groups with exactly p conjugacy classes were constructed by Ivanov 
as limits of hyperbolic groups (although hyperbolicity was not used explicitly) (see [31, 
Theorem 41.2]). Ivanov group G is infinite periodic 2-generator group, in contrast to the 
Osin group, which is torsion free. The Ivanov group G is also a simple group. The proof 
(kindly explained to us by M. Sapir) is the following. Denote by a and h the generators 
of G described in [31, Theorem 41.2]. In the proof of Theorem 41.2 on [31] it was shown 
that each of elements of G is conjugate in G to a power of generator a of order s. Let us 
consider any normal subgroup of G. Suppose 7 G A^. Then 7 = ga'^g~^ for some g E G 
and some s. Hence, = g^^^g G A^ and from periodicity of a, it follows that also a E N 
as well as G A^ for any k, because p is prime. Then any element h of G also belongs to 
A^ being of the form h = ha^{h)~^, for some k, i.e., N = G. Thus, the group G is simple. 
The discussion can be completed in the same way as in the case of Osin group. 

Example 11.22. In paper [24], Theorem HI and its corollary, G. Higman, B. H. Neu- 
mann, and H. Neumann proved that any locally infinite countable group G can be embed- 
ded into a countable group G* in which all elements except the unit element are conjugate 
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to each other (see also [36]). The discussion above related Osin group remains valid for 
G* groups. 

Let us remark that almost polycyclic group are residually finite (see e.g. [35, 5.4.17]) 
while the groups from these counterexamples are not residually finite, as it is clear by 
definition. That is why we would like to complete this section with the following question. 
Question. Suppose G is a residually finite group and is its endomorphism with finite 
i?(0). Does i?(0) equal 5/(0)? 

The results of the present section are generalized to the case of coincidences in [15]. 

12. Baumslag-Solitar groups B{l,n) 

In this section based on (a part of) [12] it is proved that any injective endomorphism 
of a Baumslag-Solitar group 5(1, n) has infinite Reidemeister number. In [12] the similar 
result for automorphisms of B{m,n) is obtained as well. 

Let B{l,n) = {a,b : a~^ba = b"',n> 1) be the Baumslag-Solitar groups. These groups 
(see [8]) are finitely-generated solvable groups (in particular amenable) which are not 
virtually nilpotent. These groups have exponential growth [5] and they are not Gromov 
hyperbolic. Furthermore, these groups are torsion free and metabelian (an extension of 
an Abelian group by an Abelian). More precisely one has 

Proposition 12.1. B{l,n) = Z[l/n] Z, where the action of Z on Z[l/n] is given by 
9{l){x) = x/n. 

Proof. The map defined by L{a) = (0, 1) and L{b) = (1,0) extends to a unique homomor- 
phism L : B{l,n) Z[l/r?,] x Z, because 

L{a~^) * L{b) * i(a) = (0, -1) * (1, 0) * (0, 1) = (0, -1) * (1, 1) = (n, 0) = i(&"). 

One has 

(28) iia^b^a-"-) = (0, r) * (s, -r) = o) . 

The map l is clearly surjective. Let us show that this homomorphism is injective. 
Let us remark that the group relation implies a~^b^^a = Hence for any s one has 
a~^b^a = h^^. Thus we can move all to the right (until they annihilate with some a or 
take the extreme right place) and all a to the left, with an appropriate changing of powers 
of b. Hence any word in 5(1, n) is equivalent to a word of the form w = a^^VaJ''^., where 
ri > 0, r2 < 0. Then l{w) = (m, ri +r2) for some m G Z[l/n]. Hence, if ri +r2 7^ 0, then 
l{w) 7^ e. Let now ri + r2 = 0, then by (28) if l{w) = e, then s = and w = e. □ 

Consider the homomorphism | \a : B{l,n) — > Z which associates to each word w G 
B{1, n) the sum of the exponents of a in this word. Since this sum for the relation is zero, 
this is a well defined map, which is evidently surjective. 

Proposition 12.2. We have a short exact sequence 

— > K — > B{l,n)^-^Z — > 1, 
where K is the kernel of \ \a. Moreover, B{l,n) equals a semidirect product K x Z. 



Proof. The first statement follows from surjectivity of | ■ !„. Since Z is free, this sequence 
splits. □ 
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Proposition 12.3. The kernel K coincide with the normalizer N (b) of the subgroup {b) 
generated by b in B{l,n): 

(29) — ^ N{b) — > B{l,n)^-^Z — ^ 1. 

Proof. We have N{b) C K. The quotient B{l,n)/N{b) has the following presentation: 
a~^ba = b , b = 1. Therefore this group is isomorphic to Z under the identification 
[a] ^ Iz- Hence the natural projection coincides with the map | \a and we have the 
following commutative diagram 

N{b) 5(1, n) B{1, n)/{N{b)) 1 

n = s 

K B{1, n) Z 1. 

The five-lemma completes the proof. □ 

Proposition 12.4. Any homomorphism cf) : 5(1, n) B{l,n) is a homomorphism of the 
short exact sequence (29). 

Proof. Let (p be the homomorphism induced by on the abelianization B{l,n)ab of 
B{1, n). The group 5(1, n)ab is isomorphic to Zn-i + Z. The torsion elements of B{l,n)ab 
form a subgroup isomorphic to Zn-i which is invariant under any homomorphism. The 
preimage of this subgroup under the projection 5(1, n) B{l,n)ab is exactly the sub- 
group N{b), i.e. the elements represented by words where the sum of the powers of a is 
zero. So it follows that N{b) is mapped into N{b) and the result follows. □ 

Theorem 12.5. For any injective homomorphism of B{l,n) the Reidemeister number 
is infinite. 

Proof. By Proposition 12.4 it is a homomorphism of short exact sequence. The induced 
map (j) on the quotient is an injective endomorphism of Z. If = Idz, then by (2) the 
number of Reidemeister classes is infinite. Hence, is multiplication by fc 7^ 0, 1. But this 
is impossible. Indeed, when we apply to the relation a~^ba = 6", under the identification 
of Proposition 12.1 l : 5(1, n) = Z,[l/n] xi Z we have, because 0(6) G N{b) and hence 
i(0(6)) = (rf,0) for some d E Z[l/n], 

{nd, 0) = i(0(6")) = L{(f){a~^ba)) = i{a~''<i){b)a^) = {d ■ n^ 0). 

This implies that either n^"'^ = 1 or 0(6) = 0. □ 
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